Visible light communication system with pixel alignment for high data rate

ABSTRACT

A visible light communication system has a two-dimensional array of sources and an image sensor with a pixel grid defining a pixel direction (which in some embodiments is a rolling shutter direction). Modulation of the light flashes the sources to define a series of one-dimensional patterns encoding the data to be transmitted. The patterns extend in a direction that projects onto the imager orthogonal to the pixel direction. In the case of a rolling shutter, the pattern is orthogonal to the rolling shutter direction, which allows the encoding of multiple bits per line of the image frame. In other cases, the orthogonal direction obtains the maximum resolving power available from the image sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a divisional application of co-pending U.S. application Ser. No.15/975,033, filed May 9, 2018, issued as U.S. Pat. No. 10,348,404 onJul. 9, 2019, which is incorporated herein by reference in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

Not Applicable.

BACKGROUND OF THE INVENTION

The present invention relates in general to visible light communication(VLC) sending digital data, and, more specifically, to optimizingalignment of generated light patterns to a pixel array of an imagesensor to increase parallel data transmission capabilities.

Visible light based communication (VLC), also referred to as LiFi, is awireless data communication technology being actively researched forautomotive applications and for consumer electronics applications. Datatransmission involves modulating (i.e., flashing) a light source such asa light emitting diode (LED) to encode data, and receiving the modulatedlight at a light sensor such as a photodiode or a camera to decode thedata.

A vehicle having a VLC receiver might receive VLC signals from a fixedsource (e.g., an LED traffic light) or from a mobile source (e.g., anLED signal light on another car). The data being shared may be relatedto traffic information or control, hazard warnings, navigationassistance, and many other types of data. A preferred image sensor is a“camera on a chip” comprising a two-dimensional array of pixels forcapturing successive image frames taken at a rate that can distinguishthe flashing of the light source. A camera with a wide field of view isdesirable in order to detect and track a VLC image source, or evenmultiple sources simultaneously. A typical VLC transmitter uses asingular LED or an array of LEDs acting in unison. To increase the rateat which VLC data can be exchanged, individual LEDs or groups of LEDscan be modulated independently to provide parallel bit streams in thedata transmission (e.g., taking advantage of a rolling shutter, asexplained below). The number of separate streams based on a transmitterLED array and a receiver pixel array depends on various opticalcharacteristics (separation distance, field of view, numbers of LEDelements in the arrays, relative motion, exposure duration per pixelrow, etc.) which determine the number of separate regions that can begenerated by the LED array that fall within the resolution (resolvingpower) of the image sensor.

Complementary metal-oxide semiconductor (CMOS) image sensors areparticularly advantageous since they provide good image quality with lowpower requirements, are low cost, and are often already present on avehicle as an object detection sensor for other vehicle systems (e.g., alane departure monitor). CMOS image sensors are also common on othertypes of devices which may be used as VLC receivers, such assmartphones.

A CMOS imager utilizes an image read-out process known as a rollingshutter, wherein the image exposure and read-out functions are conductedon a row-by-row basis (i.e., the rows of pixel are converted into adigital signal one row at a time). As used herein, the terms “row” and“column” are used interchangeably since CMOS sensors are available indifferent configurations that handle lines of pixels from top to bottomof an image and from side to side. Moreover, the CMOS sensor could beplaced on mounted to a printed circuit board (PCB) oriented in anyorthogonal direction, and the camera containing the PCB could beattached to the vehicle such that the row and column directions havebeen rotated to any orthogonal direction (based on the mountingrequirements). The use of a rolling shutter results in a temporalaliasing, wherein the image's pixel row/columns include a slight timedelay that may capture artifacts in moving objects or changes inlighting levels in the scene since different rows within a single imageframe will capture the same object at slightly different times. Thisproperty of the rolling shutter has been used to increase the data rateof a VLC transmission by flashing the LED source at a frequencycorresponding to the exposure times of successive rows (requiring thatthe LED source spans a plurality of the pixel rows in the camera). Theresulting image of the LED source consequently displays alternatingbands of light and dark lines which encode successive bits in a serialdata stream.

It would be desirable to further increase data transmission speeds witha robust, reliable system that maintains low cost and which enables moredata-hungry applications.

SUMMARY OF THE INVENTION

In one aspect of the invention, a light communication system comprises atransmitter modulating an array of light sources to encode data and areceiver with a two-dimensional array of pixels defining rows andcolumns. The pixel array images the sources to receive the data using arolling shutter in a predetermined direction. The transmitter andreceiver exchange performance characteristics after the receiver detectsmodulation of the sources. The transmitter further modulates the sourcesto encode bits of the data in adjacent sub-blocks of the sources alignedin a one-dimensional pattern extending orthogonal to the predetermineddirection as received by the receiver. In one preferred embodiment, thetransmitter adjusts the alignment of the one-dimensional patternaccording to the identification of the predetermined rolling shutterdirection.

According to another aspect of the invention, a method for lightcommunication includes arranging an imager to receive light from anarray of sources, wherein the imager has a pixel grid defining a pixeldirection. Light from the sources to the imager is modulated by flashingaccording to a series of one-dimensional patterns encoding data, eachone-dimensional pattern extending in a direction that projects onto theimager orthogonal to the pixel direction. An image is read-out from theimager resulting in a two-dimensional pattern encoding the data. Therespective patterns from a plurality of corresponding pixel rows in thepixel grid are identified, and the patterns are decoded to recover theencoded data.

According to yet another aspect of the invention, a light communicationmethod comprises arranging an imager to receive light from an array ofsources. The array of sources is arranged in a two-dimensionalorthogonal grid according to a grid direction. The imager has a pixelgrid defining a pixel direction. A misalignment between the pixeldirection and the grid direction is determined. Thereafter, light fromthe array of sources to the imager is modulated by flashing according toa series of one-dimensional patterns encoding data. Each one dimensionalpattern extends in a slanted direction on the two-dimensional grid sothat the series of one-dimensional patterns project onto the imagerorthogonal to the pixel direction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a conventional VLC system.

FIG. 2 shows an example of a video frame captured by an image sensorwith multiple light sources visible in the frame.

FIG. 3 shows a video frame in which a VLC source array is visiblewherein the resolving power of the image sensor is sufficient todistinguish a plurality of separate sub-blocks of sources in the sourcearray.

FIG. 4 is a block diagram of a portion of a CMOS image sensor.

FIG. 5 is a timing diagram showing exposure times and read-out times forvarious rows in a rolling shutter process.

FIGS. 6 and 7 are plan views showing elements of an LED array which ismodulated between ON and OFF states, respectively.

FIG. 8 shows a portion of an image frame resulting in alternating lightand dark bands in an image taken of the LED array of FIGS. 6 and 7 usinga rolling shutter.

FIGS. 9 and 10 are plan views showing elements of an LED array which aremodulated in accordance with the invention to include a one-dimensionalpattern extending orthogonal to a predetermined row/column direction ofthe image sensor.

FIG. 11 shows an image taken of the LED array of FIGS. 9 and 10 using arolling shutter, resulting in alternating pixel regions within eachimage row according to the one-dimensional pattern.

FIG. 12 shows an illuminated LED array wherein an image field is showncorresponding to a row of an image sensor when the alignment of theone-dimensional pattern is not orthogonal.

FIG. 13 is a block diagram showing a VLC system of the present inventionwith two-way communication.

FIGS. 14A-14C show a time changing illumination pattern from an LEDarray, and FIG. 14D shows a resulting rolling-shutter image frame.

FIG. 15 shows an LED source array generating a one-dimensional datapattern aligned with a vertical axis and illuminating an image sensorwhich is oriented such that a vertical pixel column lies at an obliqueangle to the line of the one-dimensional pattern.

FIG. 16 shows a grid corresponding to a pixel array of the image sensorof FIG. 15 and the locations where the image of the pattern of FIG. 15falls on the image sensor (i.e., making an angle corresponding to apixel direction of the image sensor).

FIG. 17 shows an LED array generating a line of source sub-blocksrepresenting different bit streams, wherein the line is oriented at apredetermined angle with respect to the horizontal and vertical axes ofthe LED array.

FIG. 18 shows a grid corresponding to the pixel array of FIG. 16 whenimaging the sub-blocks as generated in FIG. 17.

FIG. 19 is a flowchart showing one preferred embodiment of theinvention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The invention provides improved rates of data transfer as a result ofachieving particular alignments between the row/column directions of thetransmitting array of light sources and the row/column directions of thereceiving array of image sensor pixels. For example, a two-dimensionalarray of sources and an image sensor are spatially arranged so thatmodulated light from the source array is received by the sensor. Theimage sensor has a pixel grid defining a pixel direction (which in someembodiments is a rolling shutter direction). The flashing defines aseries of one-dimensional patterns encoding the data to be transmitted.The patterns extend in a direction that projects onto the imagerorthogonal to the pixel direction. In the case of a rolling shutter, thepattern is orthogonal to the rolling shutter direction, which allows theencoding of multiple bits per line of the image frame. In other cases,the orthogonal direction obtains the maximum resolving power availablefrom the image sensor.

FIG. 1 shows a conventional visible light communication system having aVLC transmitter 10 and a VLC receiver 15. Transmitter 10 includes anencoder 11 which receives data 12A to be transmitted, and which drivesan LED array 13 to emit a flashing VLC light signal according to theencoded data. LED array 13 may be part of a dual purpose light sourcewhich adds the VLC function to a traffic light, vehicle running light, aLCD/OLED display of a mobile device (e.g. a cell phone), fixed displayor signage, or other types of artificial lighting applications. Visiblelight 14 projected from LED array 13 flashes at a high rate which isundiscernible to the human eye but which carries data to a lightdetector 16 (e.g., a camera) in receiver 15. A source detector andtracker 17 receives a succession of image frames from camera 16, anduses known techniques for identifying any transmitting VLC sources andextracting the flashing signals inherent in the collected frames. Theflashing information is then processed by a decoder 18 which recoversdata 12B as a copy of the original data 12A sent by transmitter 10.

FIG. 2 shows a sample image taken by a receiving camera which mayinclude more than one potential source of VLC signals. A traffic light21 includes an LED array 22 as part of the production of trafficsignals, and a vehicle 23 includes an LED array 24 as part of a brake ortaillight of vehicle 23. A typical image my include other natural orartificial light sources that need to be examined as to whether theycarry VLC signals, such as the sun 25. The source detector and trackeruses conventional techniques in order to identify objects in the imageshaving the characteristic flashing of standard VLC signals, and then thecorresponding objects are inspected to extract the standard VLC signals.Depending upon the distance between a receiving camera and an array ofVLC sources and other factors, it may be possible to resolve individuallight sources or sub-groups of light sources within a detected sourcearray. Based on the resolving power of a typical camera and on the sizeof individual LEDs in a typical LED array, it would usually not bepossible for the camera to resolve an individual LED. However, the VLClight source may typically have a large number of LEDs arranged in atwo-dimensional grid, and the full grid may have an apparent size on thecamera that spans many rows and columns of pixels in the camera. Asshown in FIG. 3, an array 27 is seen within an image frame 26 at asufficient resolving power that a sub-group of LEDs 28 can be separatelyresolved. Therefore, array 27 could potentially be used to generateseveral data streams in parallel for broadcasting to the receivingcamera.

FIG. 4 shows a portion of a typical CMOS camera 30. A top row of pixels31 has individual pixels 31A, 31B, 31C, and 31D. Rows 32, 33, and 34 arearranged beneath row 31, so that all the individual pixels fall intocolumns A, B, C, and D. A set of Row Select lines each respectivelyconnects to the pixels of a respective row. Output lines from eachrespective pixel of the same column are directly connected to a columnmultiplexer 35 in common. An output from column multiplexer 35 isamplified through a buffer amplifier 36 to provide readout of imageframes from camera 30 as known in the art. A particular Row Select lineis activated during the time that a particular row is read out, so thatonly the outputs of that row are sent to column multiplexer 35 at anyone time.

Operation of the rolling shutter process in a CMOS camera is shown inFIG. 5. Each respective row has its own respective image integrationtime and readout time which are staggered with respect to the otherrows. Thus, a first pixel Row 1 has an image integration time 37 and areadout time 38. Some rows can be integrating an image at the same timebut all rows must have separate readout times. Thus, Row 2 has anintegration time 40 followed by a readout time 39. Collection of a firstimage frame (Frame 1) ends after the last row has been read out. Then asubsequent frame (Frame 2) begins with integrating an image in Row 1during image integration time 41, which is followed by a readout time42.

Using the separate image integration times and readout times fordifferent rows that occur in a rolling shutter camera, it has been knownto transfer VLC data at a rate higher than the frame rate of a CMOScamera by alternately flashing the LED light source to be ON or OFFduring the staggered times that respective pixel rows are activelyintegrating an image (the ON/OFF flashing or other types of variationwithin the LED output can encode the data according many known types ofmodulation). As shown in FIG. 6, an LED array 45 with individual LEDs 46may be on at a first instant of time t₁. At another instant of time t2shown in FIG. 7, LEDs 46 within array 45 are turned off. By controllingthe ON and OFF flashing to repeat at a rate faster than the amount oftime required for reading all of the rows covering the light source,light and dark bands 48 appear in a single image frame 47 as shown inFIG. 8. In this example, the rolling shutter moves from top to bottom sothat the LED array is shown at a slightly later time moving downward inthe resulting image frame. Light and dark bands 48 can be converted to adigital signal 49 that transitions between high and low logic levels.Digital signal 49 can be interpreted as an On/Off keying signal to passbinary information across the resulting image in which the rows encodethe information directly. Other keying strategies can also be usedincluding color based, frequency based, pulse position modulation (PPM),and pulse amplitude modulation (PAM) as known in the art, and all suchstrategies can be utilized in the present invention.

With the technique of FIG. 6-8 for treating each row of a CMOS imagesensor in a rolling shutter frame to collect a corresponding data point,the data transmission rate is increased by a factor determined by thetime between integration of successive rows. The data transfer remainsto be a single “bit stream” since all the LEDs in the transmitting arrayare turned On and Off in unison, even though the CMOS image sensor iscapable of resolving smaller sub-blocks of the LED sources. Operation inunison has ensured that the bands encoding the bit stream are reliablycreated in the camera images. One embodiment of the present inventionobtains a further increase in the data transmission rate by encodingmore than just a single information “bit” into each row of the rollingshutter frame. The ability to achieve this increase is obtained by usinga plurality of sub-blocks within the LED array to generate separatelyresolvable bit regions by the pixels of the CMOS sensor, while furtherimposing a condition that a one-dimensional On/Off pattern generated bythe bit regions extends in a direction that has a particular orientationwith respect to a predetermined row/column direction of the CMOS sensor.When the CMOS sensor uses a rolling shutter, then the direction of theone-dimensional pattern extends orthogonal to the direction of therolling shutter rows of pixels. For example, if the CMOS sensor'srolling shutter direction is aligned orthogonally to the row/columnlayout of the LED array, and if the CMOS sensor and LED array arepositioned at a optimal distance based on the camera's lens opticalcharacteristics, LED array size, and pixel size, then it becomespossible to utilize all of the camera's pixels to transmit data at muchfaster data transmission rates.

Orthogonal pattern generation for a preferred embodiment is shown inFIGS. 9-11. An LED array 50 includes a plurality of source (e.g., LED)sub-blocks 51 arranged in a two-dimensional grid of rows and columns.Depending upon the distance between the transmitter and receiver,sub-blocks 51 may each comprise a cluster or region of individual LEDswith sufficient area that each sub-block 51 is separately resolvable bythe receiving image sensor. In the event that a camera is sufficientlyclose to LED array 50 that it can resolve a single LED, then eachsub-block 51 may be comprised of a single LED. For increasing the dataencoded in each row of a received image, the flashing of sub-blocks 51which modulates to encode bits of the transmitted data is arranged in aone-dimensional pattern extending horizontally. Thus, the patternchanges from left to right and is constant from top to bottom alongcolumns 52 and 53 (i.e., the sub-blocks in the columns are turned on andoff in unison). The number of rows from top to bottom preferablycorresponds to the full usable size of array 50 since it maximizes theamount of data that can be sent during one frame of the image sensor.FIG. 9 shows the pattern at a first instant in time. FIG. 10 shows thepattern at a second instant of time when sub-blocks of array 50 exhibita different one-dimensional pattern extending horizontally.

In a preferred embodiment with a rolling shutter, the CMOS image sensorcollects an image of source array 50 such that the direction of therolling shutter proceeds vertically from the top to the bottom of array50. Therefore, the direction of the one-dimensional pattern (horizontal)is orthogonal to the rolling shutter direction (vertical) in which therow-by-row readout of an image occurs. As a result, a complete imageframe of array 50 collected by the image sensor has the appearance shownin FIG. 11 (there may be an offset between data packets and images asknown in the art). Thus, an image frame 55 of the LED array includesindividual pixels 56 wherein each horizontal row of pixels correspondsto a distinct “snapshot” of the LED source array. A respectiveone-dimensional pattern used in encoding the data by the LED sourcearray appears in each horizontal row. The frame data is encoded in ahorizontal direction x along each image row and encoded in the verticaldirection for rows collected at a different time t resulting from therolling shutter. The ON/OFF states of the pixels in each row are used togenerate a digital signals 57 and 58 for the indicated rows in thisexample of the encoding.

In order to reliably distinguish each separate parallel data bit regionencoded in the one-dimensional patterns of FIGS. 9 and 10, the alignmentof the one-dimensional pattern direction should be reasonably close toorthogonal to the rolling shutter direction of the CMOS image sensor.FIG. 12 shows an image capture area 59 of a pixel row of the imagesensor which is collected using a rolling shutter, wherein there is amisalignment between the top-down and side-to-side axes of the grids ofthe LED array and the image sensor. Misalignment may result from normalchanges in road surface conditions, other relative movements betweentransmitter and receiver, inherent differences in the design orinstallation of components, rolling motion, differences in axialorientations, and image distortions resulting from optical errors,translation, or image perspective (i.e., not viewing the LED array faceon). For example, some CMOS sensors are made with a vertical rollingshutter and some are made with a horizontal rolling shutter, so that atransmitting array that only generated the patterns in one directioncould not properly align with CMOS sensors of both types (and even if itcould, issues related to perspective, optical distortion, andtranslation could still create problems for resolving the patterns).

Depending on the severity of misalignment, the image that would resultfor a pixel row having misaligned capture area 59 could fail tocorrectly exhibit the desired ON/OFF pattern. For use in situationswhere the desired alignment could not be automatically ensured, thepresent invention introduces two-way communication between a VLCtransmitter and a VLC receiver in order to orient the transmitted imagesfrom the LED array such that the desired alignment is maintained. Sincethere would typically be a large number of individual LEDs included ineach sub-block of the generated data pattern (due to the sizes of theexpected LED arrays and the average distance between the transmitter andreceiver), it would usually be feasible to alter the direction of theone-dimensional pattern generated on the LED array to almost anyarbitrary direction while maintaining accurate reproduction of thepattern.

As shown in FIG. 13, a VLC transmitter 60 includes a VLC transmitcontroller 61 and an LED array 62. In this embodiment, a receiver 63 inVLC transmitter 60 forms part of a return channel between VLCtransmitter 60 and a VLC receiver 64. VLC receiver 64 includes a CMOSimage sensor or other detector 66 coupled to a VLC receive controller65. A transmitter 67 connected to VLC receive controller 65 formsanother portion of the return channel with receiver 63 using anyconvenient communication technology, such as a VLC return channel or anRF-based channel (e.g., Wi-Fi) as known in the art. Using the returnchannel together with the main VLC signal channel between LED array 62and detector 66, transmitter 60 and receiver 64 mutually exchangeperformance characteristics which are sufficient to ensure the desiredalignment of the one-dimensional flashing pattern with the camera'spixel direction (e.g., rolling shutter direction). In addition, otherperformance characteristics are exchanged which can be used to operateat the maximum available optical resolution and to improve other aspectsof image quality. The performance characteristics can include one ormore of the following characteristics: number of light sources (i.e.,LEDs), arrangement of light sources, number of receiver pixels,arrangement (rows and columns) of receiver pixels, localization andfrequency of data packet transmission errors determined by errorchecking from prior decoded two-dimensional data pattern (e.g., CRCchecks), nominal exposure time for a pixel row of the receiver, apparentsize of a sub-block of a modulated group of light sources as detected bythe receiver, relative orientations of the transmitter and receiver,relative translation of the transmitter sources and receiver pixels,relative perspective translation of the transmitter and receiver,identification of the rolling shutter direction, and relative movementbetween the transmitter and receiver.

FIG. 14 shows a further example of the invention wherein an LED array 70includes individual LEDs 71 extending in a mainly horizontal array ofthe type that may be present in a vehicle brake light/taillightassembly, for example. Separate sub-blocks 72-77 are uniquely driven inorder to generate a one-dimensional pattern for flashing the LEDsaccording to encoded data for transmitting to a VLC receiver. Eachsub-block 72-77 is shown to include a grouping of four individual LEDs.The other LEDs in array 70 are not flashed at a VLC frequency, therebyallowing sub-blocks 72-77 to be separately identified and tracked. InFIG. 14A, a one-dimensional pattern is shown having selected sub-blocksof LEDs being on and off at a first time t₁, (with cross-hatching andlack of cross-hatching representing On and Off states). The time t₁persists long enough to cover a major fraction of an image integrationtime for at least one rolling shutter row that is being exposed toregister the data signal on that row and short enough to expose no morethan a few rolling shutter rows based on the logic of calculationresulting from the performance characteristics previously transmitted.FIGS. 14B and 14C show different On/Off one-dimensional patterns duringsubsequent image integration time for subsequent rows (or groups ofrows) of the rolling shutter image sensor. FIG. 14D shows a capturedimage frame depicting the LED array wherein respective image pixelregions 79 have captured the On/Off values of the successiveone-dimensional patterns.

In the example of FIG. 14, a lack of height in the LED array limits theability to transmit the one-dimensional pattern according to a differentdirection. However, other arrays with higher numbers of rows and columnsof individual LEDs can achieve greater flexibility in the orientation ofthe one-dimensional pattern (see, for example, FIG. 17).

An ability to align a one-dimensional pattern on the transmitting LEDarray can be useful not only for improved data rates with a rollingshutter. Improvements can be obtained with other image sensors, such asa CCD imager using a global shutter (i.e., all pixels of an image beingexposed over the same span of time). More specifically, the maximumresolving power of an imager can be obtained when the one-dimensionalpattern aligns with either of the orthogonal pixel directions (row orcolumn) because such an alignment avoids spatial aliasing problems.Referring to FIG. 15, an LED array 80 has many rows and columns ofindividual LEDs 81. A one-dimensional pattern 82 is illustrated beinggenerated vertically by LED sub-blocks in array 80. An image sensor hasa grid of pixels 83 imaging LED array 80 from an orientation such thatthe vertical directions of the respective grids form an misalignmentangle 84. In a captured image obtained by the image sensor with theorientations as shown in FIG. 15, one-dimensional pattern 82′ falls at anon-orthogonal direction on the captured image as shown in FIG. 16.Based on the size and separation of the sub-blocks in pattern 82′ andbecause of the misalignment, the sub-blocks cannot be separatelyresolved. On the other hand, after an exchange of performancecharacteristics between the transmitter and receiver, the misalignmentbetween the respective grids can be determined and then the transmittercan orient the direction of the one-dimensional pattern to be orthogonalto a pixel direction (i.e., aligned in parallel with the other pixeldirection) of the camera. Thus, LED array 80 generates a slantedone-dimensional pattern 85 shown in FIG. 17. When slantedone-dimensional pattern 85 is captured by the image sensor, theresulting image in pixel grid 83 shown in FIG. 18 displays aone-dimensional pattern 85′ that aligns along one of the orthogonaldirections of grid 83 (either horizontal or vertical can be used). Sincealiasing problems are avoided, the same size of LED sub-blocks can beclearly resolved in FIG. 18. In order to identify any discrepancybetween the grid axes, the present invention may utilize a two-wayinitialization procedure wherein the transmitter flashes patterns havingdefault sizes and orientations which are analyzed in the receiver tocalculate various performance characteristics such as resolvablesubgroup size and relative orientations. Another method to diagnoseincorrect performance characteristic settings may be through the use ofregularly spaced parity bits that are used for error correction atanother location and time than the error correction bit. Such a methodis well known, such as in the use of QR codes and may be applied here.

A preferred method of the invention is shown in FIG. 19. Beforecommencing VLC communication, a receiver mounted on a moving vehiclecaptures successive image frames (i.e., video) in step 90. Thesuccessive images are analyzed (e.g., filtered) to identify any activeVLC sources. A check is performed in step 91 to determine whether anyVLC sources are undetected. If not, then a return is made to step 90 tocontinue to monitor for a VLC source. Once a source is detected, thenserial data is received from the detected VLC transmitter in step 92.For example, the VLC transmitter may initially be sending a VLC beaconsignal using an entire LED array (or large part of it) in order toinitiate communication with a receiver. Once the beacon signal and/orserial data is received, the VLC receiver may contact the transmittervia the return channel. A mutual exchange of performance characteristics(e.g., based on appearance of the VLC source in the capture images) isperformed in step 93. In step 94, the transmitter and receiver determineand share image resolution, apparent size, relative orientation of theirrespective grids, image intensity, distortion, and other aspects of theperformance characteristics.

Based on an analysis of the performance characteristics, a determinationis made in step 95 whether parallel communication using aone-dimensional pattern of parallel subgroups of LED sources ispossible. If parallel communication is not feasible, then serialcommunication continues to be used in step 96. Then a check isperiodically performed in step 97 to determine whether the LED array isstill visible. If not, then the method ends at step 98. If the LED arrayis still visible, then a return is made to step 93 to re-perform theexchange of performance characteristics and to re-determine whetherparallel communication has become possible.

When parallel communication is possible, then parallel data regions(sub-blocks of an identified area on the LED array) are generated instep 100 such that the sub-blocks form a one-dimensional pattern alongan orthogonal camera direction. In the case of a rolling shutter CMOSsensor, then the orthogonal camera direction is perpendicular to thedirection of a rolling shutter row (wherein the pixels of a row areexposed simultaneously). It should be noted that in the presentinvention, a camera sensor with a rolling shutter does not require timesynchronization and that the time varying one-dimensional horizontaldata pattern may appear in one image or the next. It should be furthernoted that like VLC described in prior methods, an error correctionmethod must be implemented to account for errors in receiving ordecoding the correct data or loss of data through factors suchframe-to-frame time gaps. A method to regularly vary parity check bitsfor error correction would occur in both time and the horizontalone-dimensional data direction. In a non-rolling shutter (i.e., globalshutter) image sensor, the light source array could vary the pixelintensities across both horizontal and vertical directions to create atwo dimensional grid-like pattern in which the display of one patternwould be timed to coincide with the global shutter exposure. This timesynchronization may occur via methods known in prior art. As usedherein, the term “orthogonal direction” for a global shutter imagesensor means either one of the orthogonal rows and columns of thesensor. Furthermore, while the actions discussed above for aligning thepattern direction with a desired pixel direction of the camera utilizean electronically performed adjustment of the mapping of data regions(sub-blocks) onto the LED array, it is also possible to achieve adesired alignment by mechanical re-orientation of either the LED arrayor the image sensor (e.g., by mounting on a gimbal driven by aservomechanism). Of course, fixed permanent mounting of a transmitterand receiver could also ensure the desired orthogonal alignment in orderto practice the present invention.

In order to improve image quality and to shorten the time betweenconsecutive frames, subsequent images may be captured usingsub-windowing in step 101. Sub-windowing refers to reading out of animage from the image sensor using only a portion of the full sensorpixel grid. The VLC source is tracked within several image frames, andthen image collection is performed for a series of subsequent images tocollect data only within the sub-windowed area. Periodically, fullimages may be captured in order to continue to correctly track thelocation of the VLC source array.

In step 102, other image corrections can be applied to captured images.For example, various distortions may be present in an image besides thediscrepancy in vertical orientation of the respective grids. Based onanalysis of a default pattern sent during initialization, compensationmay be applied to either the pattern driven onto the transmitter arrayor to the captured image to reverse the distortion. Using the correctedimages, data signals are extracted in step 103 in a known manner inorder to reconstruct the originally encoded data.

The foregoing invention achieves higher data transfer speeds forlight-based communication based on displaying an optical pattern from anLED array such that the light pattern is orthogonal to the rollingshutter direction of the camera even though the camera may bearbitrarily oriented and positioned in space. The data transfer isdependent upon both time variation of LED output (flashing) and aone-dimensional (1-D) binary array output of the LED array (which isdetermined based on the camera's ability to differentiate more than one“pixel” or sub-blocks along the LED array). To achieve real worldoperation of this invention, a data exchange mechanism is used toprovide the LED transmitter with knowledge of the camera's perspective,position, field of view (FOV), orientation, resolution, row readouttime, distortion, and other factors to display the 1-D binary arrayoutput with a correct orientation for maximum use of the camera'scapabilities. If an LED array is sending to multiple receivers, then ituses the capabilities of the camera operating at the “least commondenominator.”

The invention utilizes a rapidly changing 1-D LED pattern in which therolling shutter effect of a CMOS based image sensor along with thetemporal variation between camera pixel rows to create an effective 2-Dpattern for encoding data. The invention provides N times more datatransmission than normal VLC communication, where N is equal to the LEDgrid's effective spatial size determined by properties of the imagesensor, lens, and operating distance. The process is far less sensitiveto correct timing from a single or multiple cameras than prior arttechniques and is far less sensitive to exposure timing and blooming.The process can support cameras of different exposure times anddifferent frame to frame gaps without modifications.

The VLC receiver's camera would preferably identify an optimal exposuretime to avoid over-exposure of the LED array and blurring of the 1-Dbinary array output. This may entail active modulation of the dataoutput frequency by the LED to match the camera's rolling shutter speedand exposure length. In addition, sub-windowing can be used to increaseimage capture and thus data transmission speeds.

The relative data transmission rate increase will depend on a number offactors including distance between objects, rolling shutter orientation,LED array size, image sensor size, camera lens optical characteristics,required exposure duration, environmental noise level, and more.Detection and communication between transmitter and receiver preferablystarts with standard VLC wherein all LEDs are operated uniformly inparallel during an initial data exchange. The initial exchangedetermines whether the transmitter's LED array is sufficiently largewithin the camera image to employ this method. This determination mayalso take into account quality of service with other VLC devices thatare also communicating with the same transmitter. Data exchange includessensor characteristics and LED array characteristics, including cameralens optical characteristics, rolling shutter direction, and exposureduration. This may also employ use of VLC to determine the relativeposition of the transmitter to the receiver. The camera can utilizeimage capture sub-windowing over the specific region of interestcorresponding to the LED array to increase data throughput. The LED 2-Darray transmits data by varying its intensity and/or color in anorthogonal direction to the camera's rolling shutter. The number ofunique LED “pixels” or sub-blocks depends on the capability of detectionby the receiving CMOS censor. Additionally, the direction of the 1-Dpattern for the LED sub-blocks can extend in any arbitrary direction,e.g., left-right, up-down, or some relative rotation (skew) based on theLED array and camera positions and distortion. After a certain amount oftime the initialization process may be repeated given the possibility ofthe camera and LED array shifting position or being blocked from thefield of view. Such a fallback state will be helpful in confirmingintegrity of data transfer and a correct angle of the LED lightingpatterns. At this step the LED intensity may also be altered.

Depending on the LEDs intensity, camera sensor sensitivity, and exposureduration, the captured image of the LED array may appear under or overexposed. This could result in blurry demarcation of the data encoded inthe rolling shutter image and/or poor signal-to-noise ratio. Therefore,an adaptive process of determining the optimal exposure length per eachpixel row may be desirable. For example, a sequence of images can beobtained using a range of exposures. Furthermore, error correctionparity bits or some other error correction scheme dispersed throughoutthe 1-D pattern and time may be used to monitor for changes inperformance characteristics. Specifically, the receiver may monitor thedata transmission rate over time or across the 1-D pattern to detect anincrease in error rates and resend the performance characteristics tothe transmitter to improve the data transmission. Afterwards, theoptimal exposure can be identified in the images, and the exposure timethat results in the best presentation of the data pattern as captured bythe rolling shutter would be used. Alternatively, an image sensor with ahigher dynamic range could be used.

The invention can be used not only for vehicular transportationfunctions, but also for indoor consumer electronics LIFI products andfor other types of outdoor data communication systems (e.g., cell phoneVLC transmission to a vehicle-mounted camera sensor). The invention canbe adapted to use color keying by utilizing multiple color layers.Instead of visible light, near-infrared LEDs which are not visible tohuman eye can be used (especially with a camera sensor not having an IRcutoff filter). This would permit more placement options for the LEDarray beyond those which might otherwise be allowed in view of stylingconsiderations for a vehicle.

What is claimed is:
 1. A light communication method, comprising:arranging an imager to receive light from an array of sources, whereinthe array of sources is arranged in a two-dimensional orthogonal gridaccording to a grid direction, and wherein the imager has a pixel griddefining a pixel direction; determining a misalignment between the pixeldirection and the grid direction; and modulating light from the array ofsources to the imager by flashing according to a series ofone-dimensional patterns encoding data in a plurality of parallel databit streams, wherein each bit stream is generated by flashing arespective one of a plurality of subgroups of the sources, each onedimensional pattern extending in a slanted direction with respect to thegrid direction on the two-dimensional grid so that the series ofone-dimensional patterns project onto the imager orthogonal to the pixeldirection.
 2. The method of claim 1 further comprising: reading out animage from the imager; identifying respective patterns from a pluralityof corresponding pixel rows in the pixel grid; and decoding the patternsto recover the encoded data.
 3. A light communication method,comprising: arranging an imager to receive light from an array ofsources, wherein the array of sources is arranged in a two-dimensionalorthogonal grid according to a grid direction, and wherein the imagerhas a pixel grid defining a pixel direction; determining a misalignmentbetween the pixel direction and the grid direction by exchanging initialdata between the array of sources and the imager that depends on theorientation of the pixel direction with respect to the grid direction;and modulating light from the array of sources to the imager by flashingaccording to a series of one-dimensional patterns encoding data, eachone dimensional pattern extending in a slanted direction on thetwo-dimensional grid so that the series of one-dimensional patternsproject onto the imager orthogonal to the pixel direction.
 4. The methodof claim 3 wherein the exchanging of initial data is performed using areturn channel between a first controller for the array of sources and asecond controller for the imager, and wherein the return channel useslight communication.
 5. The method of claim 3 wherein the exchanging ofinitial data is performed using a return channel between a firstcontroller for the array of sources and a second controller for theimager, and wherein the return channel uses RF communication.
 6. A lightcommunication method, comprising: arranging an imager to receive lightfrom an array of sources, wherein the array of sources is arranged in atwo-dimensional orthogonal grid according to a grid direction, andwherein the imager has a pixel grid defining a pixel direction;determining a misalignment between the pixel direction and the griddirection; and modulating light from the array of sources to the imagerby flashing according to a series of one-dimensional patterns encodingdata, each one dimensional pattern extending in a slanted direction onthe two-dimensional grid so that the series of one-dimensional patternsproject onto the imager orthogonal to the pixel direction; wherein theimager is comprised of a CMOS image sensor, and wherein the pixeldirection is comprised of a rolling-shutter direction of the CMOS imagesensor.